Target, method, and system for camera calibration

ABSTRACT

The present disclosure relates to a target, a method, and a system for calibrating a camera. One example embodiment includes a target. The target includes a first pattern of fiducial markers. The target also includes a second pattern of fiducial markers. The first pattern of fiducial markers is a scaled version of the second pattern of fiducial markers, such that a calibration image captured of the target simulates multiple images of a single pattern captured at multiple calibration perspectives.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 16/547,427, filed Aug. 21, 2019, which is a continuation ofU.S. patent application Ser. No. 15/720,979, filed Sep. 29, 2017. Theforegoing applications are incorporated herein by reference.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Cameras have become ubiquitous when it comes to recording images.Various cameras may record images on film or digitally store images as aseries of bits within a computer memory (e.g., a hard drive). Manydevices even have cameras integrated in them. For example, mobilephones, tablets, and laptop computers may include cameras.

One application for cameras is in the field of computer vision. Incomputer vision, in order to make inferences from recorded images,calibrating the camera recording the images can be important. Such acalibration can provide a correlation of the appearance of a recordedimage to the spatial layout of a physical scene. Further, calibrationcan correct for defects in the fabrication and/or assembly of a camerasensor/lens used to record images. For example, if an aperture of acamera is off-center with respect to a camera sensor, a calibration mayaccount for this (e.g., using a processor that provides a correction torecorded images such that they more accurately reflect a physicalscene).

One method of calibrating cameras includes applying the pinhole cameramodel. The pinhole camera model assumes the camera being calibrated isan ideal pinhole camera (i.e., a camera with no lenses and a point-likeaperture). Using the pinhole camera model approximation, the coordinates(e.g., in three-dimensions) of a physical scene may be mapped to aprojection on a two-dimensional plane, where the projection on thetwo-dimensional plane is represented by a recorded calibration image.The location of the pinhole aperture in the theoretical pinhole cameracan be determined based on the calibration image. Other parameters ofthe theoretical pinhole camera can also be determined (e.g., focallength). If the location of the pinhole aperture is not centered withrespect to the camera sensor, steps can be taken to account for theoff-center location of the aperture. Determining the location of thepinhole aperture and accounting for it may include calculating one ormore elements of a camera matrix based on one or more calibrationimages.

Other methods of calibration can be employed to correct for otherdefects inherent in optical design or due to fabrication/assembly, aswell. For example, one or more distortion coefficients can be calculatedbased on a recorded calibration image. The distortion coefficient may beused to account for optical non-uniformities arising due to a lens inthe camera (e.g., barrel distortions, mustache distortions, orpincushion distortions). In additional, other optical aberrations can beaccounted for using calibration (e.g., defocusing, tilting, sphericalaberrations, astigmatism, coma, or chromatic aberrations).

SUMMARY

An example calibration target may allow for a camera to be calibrated.The calibration target may include two or more panels each with a seriesof fiducial markers thereon. The panels may be angled with respect toone another and the fiducial markers may be positioned in one or morearrangements. The fiducial markers on the panels may be uniquelyidentifiable. When a camera to be calibrated captures or records acalibration image of the calibration target, multiple calibration imagesfrom different angular perspectives (e.g., different cameraperspectives) of a single panel may be simulated by a single calibrationimage of the two or more panels. This because the two or more panels areangled with respect to one another. Simulating multiple images with asingle calibration image may decrease computational resources necessaryfor calibration as well as reduce calibration time.

In a first aspect, the disclosure describes a target used forcalibration. The target includes a first pattern of fiducial markers.The target also includes a second pattern of fiducial markers. The firstpattern of fiducial markers is a scaled version of the second pattern offiducial markers, such that a calibration image captured of the targetsimulates multiple images of a single pattern captured at multiplecalibration perspectives.

In a second aspect, the disclosure describes a method. The methodincludes recording a calibration image of a target using a camera. Thetarget includes a first panel having a first arrangement of fiducialmarkers thereon. Each of the fiducial markers in the first arrangementis uniquely identifiable among fiducial markers in the firstarrangement. The target also includes a second panel, disposed at afirst angle relative to the first panel, having a second arrangement offiducial markers thereon. Each of the fiducial markers in the secondarrangement is uniquely identifiable among fiducial markers in thesecond arrangement. The first arrangement of fiducial markers matchesthe second arrangement of fiducial markers. The method also includesdetermining locations and identifications of one or more fiducialmarkers in the calibration image. In addition, the method includes,based on the determined locations and identifications, calibrating thecamera.

In a third aspect, the disclosure describes a system used forcalibrating a camera. The system includes a target. The target includesa first pattern of fiducial markers. The target also includes a secondpattern of fiducial markers. The first pattern of fiducial markers is ascaled version of the second pattern of fiducial markers, such that acalibration image captured of the target simulates multiple images of asingle pattern captured at multiple calibration perspectives. The systemalso includes a stage configured to translate or rotate the camera withrespect to the target.

In an additional aspect, the disclosure describes a system. The systemincludes a means for recording a calibration image of a target using acamera. The target includes a first panel having a first arrangement offiducial markers thereon. Each of the fiducial markers in the firstarrangement is uniquely identifiable among fiducial markers in the firstarrangement. The target also includes a second panel, disposed at afirst angle relative to the first panel, having a second arrangement offiducial markers thereon. Each of the fiducial markers in the secondarrangement is uniquely identifiable among fiducial markers in thesecond arrangement. The first arrangement of fiducial markers matchesthe second arrangement of fiducial markers. The system also includes ameans for determining locations and identifications of one or morefiducial markers in the calibration image. In addition, the systemincludes a means for calibrating the camera based on the determinedlocations and identifications.

The foregoing summary is illustrative only and is not intended to be inany way limiting. In addition to the illustrative aspects, embodiments,and features described above, further aspects, embodiments, and featureswill become apparent by reference to the figures and the followingdetailed description.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1A is an illustration of a panel of a calibration target, accordingto example embodiments.

FIG. 1B is an illustration of a panel of a calibration target, accordingto example embodiments.

FIG. 1C is an illustration of a fiducial marker, according to exampleembodiments.

FIG. 1D is an illustration of a panel of a calibration target, accordingto example embodiments.

FIG. 1E is an illustration of a calibration process.

FIG. 1F is an illustration of a calibration process.

FIG. 1G is an illustration of a calibration process, according toexample embodiments.

FIG. 2A is a front-view illustration of a calibration target, accordingto example embodiments.

FIG. 2B is a top-view illustration of a calibration target, according toexample embodiments.

FIG. 2C is an illustration of a calibration process.

FIG. 2D is an illustration of a calibration process, according toexample embodiments.

FIG. 3A is an illustration of a panel of a calibration target, accordingto example embodiments.

FIG. 3B is an illustration of a panel of a calibration target, accordingto example embodiments.

FIG. 4A is a front-view illustration of a calibration target, accordingto example embodiments.

FIG. 4B is a top-view illustration of a calibration target, according toexample embodiments.

FIG. 4C is a front-view illustration of a calibration target, accordingto example embodiments.

FIG. 4D is a top-view illustration of a calibration target, according toexample embodiments.

FIG. 4E is a side-view illustration of a calibration target, accordingto example embodiments.

FIG. 5 is an illustration of a panel of a calibration target, accordingto example embodiments.

FIG. 6A is an illustration of a panel of a calibration target, accordingto example embodiments.

FIG. 6B is an illustration of a panel of a calibration target, accordingto example embodiments.

FIG. 6C is an illustration of a panel of a calibration target, accordingto example embodiments.

FIG. 6D is an illustration of a panel of a calibration target, accordingto example embodiments.

FIG. 6E is an illustration of a panel of a calibration target, accordingto example embodiments.

FIG. 7 is a flowchart illustration of a method, according to exampleembodiments.

DETAILED DESCRIPTION

Example methods and systems are described herein. Any example embodimentor feature described herein is not necessarily to be construed aspreferred or advantageous over other embodiments or features. Theexample embodiments described herein are not meant to be limiting. Itwill be readily understood that certain aspects of the disclosed systemsand methods can be arranged and combined in a wide variety of differentconfigurations, all of which are contemplated herein.

Furthermore, the particular arrangements shown in the figures should notbe viewed as limiting. It should be understood that other embodimentsmight include more or less of each element shown in a given figure. Inaddition, some of the illustrated elements may be combined or omitted.Similarly, an example embodiment may include elements that are notillustrated in the figures.

I. OVERVIEW

Example embodiments relate to a target, method, and system for cameracalibration. A target used for camera calibration as described hereinmay reduce calibration time, reduce computing resources needed forcalibration, reduce the space needed for calibration, reduce themechanical complexity of calibration, and reduce the number ofcalibration images that need to be captured or recorded in order forcalibration to successfully be completed. Using a target as describedherein may improve calibration efficiency because the calibration targetis designed in such a way that when a single calibration image iscaptured or recorded of the calibration target, additional calibrationimages can be simulated.

For example, in some embodiments, the calibration target may includefiducial markers located thereon. The fiducial markers may be arrangedon two or more panels. Further, the fiducial markers may be arranged inpatterns on each panel. At least two of the patterns on a given panelmay be scaled versions of one another (e.g., one pattern of fiducialmarkers is a smaller square and another pattern of fiducial markers is alarger square). Such patterns can be used in a captured or recordedcalibration image of the calibration target (e.g., captured or recordedby a camera requiring calibration) as a simulation of multiple cameralocations. Rather than having to reposition a calibration targetrelative to the camera and capture or record multiple images, as may bethe case in alternate calibration schemes, a single calibration imagecan be used to calibrate various camera perspectives relative to thecalibration target (e.g., distances from the calibration target orangles relative to the calibration target). In embodiments where thecalibration includes using a pinhole camera model to calibrate thelocation of an aperture of the camera, multiple calibration images fromdifferent perspectives may be used. Thus, having a single calibrationimage that can represent multiple camera perspectives may savecalibration time, calibration space, mechanical complexity, and/orcomputational resources.

In addition to having a calibration target that can be used to simulatemultiple camera perspectives in one degree of freedom (e.g., multipledistances from the camera, i.e., z-direction), some example embodimentsinclude a calibration target where two panels are positioned at angle(s)relative to one another. In such embodiments, the arrangement of thefiducial markers on the two panels may be substantially or exactly thesame. Given the angle(s) of the panels relative to one another, one ormore additional degrees of freedom (e.g., rotation about the x-axis,y-axis, and/or z-axis) may be simulated within a single captured orrecorded calibration image of the calibration target. For example, twopanels may be rotated about ay-axis with respect to one another (e.g.,by 30°). This may allow a camera at a single perspective relative to thecalibration target to capture or record one perspective relative to thefirst panel in the calibration target and a second perspective relativeto the second panel in the calibration target. Because the two panelsmay be substantially or exactly the same (e.g., have similar, or evenidentical, fiducial markers arranged thereon), the single calibrationimage can be used to calibrate for two angular perspectives relative tothe panels. Additional embodiments may include calibration targets thathave additional angles between panels and/or fiducial-marker patternshifts to simulate additional degrees of freedom in captured or recordedcalibration images of the respective calibration target.

Additionally, the fiducial markers on the calibration targets/panels maybe designed such that a location on the fiducial marker (e.g., thecenter of the fiducial marker) and an identity of the fiducial marker(e.g., name or ID of the fiducial marker) can be identified (e.g., by ahuman reviewing a captured or recorded calibration image and/or by aprocessor analyzing a captured or recorded calibration image to performcamera calibration). In order to accomplish this, one example embodimentincludes fiducial markers that each have four distinct regions. Oneregion may be a crosshair region used to pinpoint a location of thecenter of the fiducial marker. A second region may be a fiducial markeridentification section that can be used to identify the fiducial markeris (e.g., the ID of a given fiducial marker). In some embodiments, thesecond region may be an angular barcode section that has a series ofangular bits used to represent the ID of the fiducial marker as abarcode. A third region of a fiducial marker may be a fiducial boundingsection that can be used to identify an edge/boundary of the fiducialmarker, as well as to determine whether a fiducial marker is actuallypresent in a calibration image (e.g., as opposed to noise in acalibration image). Lastly, a fourth region of a fiducial marker may bea human-readable label that may readily indicate an ID of the fiducialmarker to a human. In some embodiments, fewer or greater than fourregions may be included in a fiducial marker. For example, in someembodiments, fiducial markers may include only the crosshair region anda fiducial bounding section. Additionally, in some embodiments, variousfiducial markers across the same embodiment may include differentsubsets of the four regions described above (e.g., one fiducial markermay include all four regions described above while another fiducialmarker may include only the crosshair region).

In some embodiments (e.g., embodiments where two or more panels of acalibration target have a similar or the same arrangement of fiducialmarkers thereon), one or more of the fiducial markers on each panel maybe used as a panel-identification fiducial marker. Apanel-identification fiducial marker can be used during calibration(e.g., during image analysis of a calibration image) to determine whichpanel, of a number of panels having similar appearances, is currentlybeing analyzed. Other methods of panel identification are also possible.

II. EXAMPLE SYSTEMS

The following description and accompanying drawings will elucidatefeatures of various example embodiments. The embodiments provided are byway of example, and are not intended to be limiting. As such, thedimensions of the drawings are not necessarily to scale.

FIG. 1A is an illustration of a first panel 102 of a calibration target,according to example embodiments. The calibration target may be used inthe process of calibrating one or more cameras. For example, thecalibration target may be used to capture or record one or morecalibration images. The captured or recorded calibration images may thenbe used to determine features about the camera (e.g., distortioncoefficients, camera matrices, lens position/orientation, camera sensorposition/orientation, aperture position/orientation, etc.). Once thefeatures about the camera are determined (e.g., using a pinhole cameraapproximation model), any defects or irregularities may be accountedfor. Accounting for defects or irregularities may include modifying theaperture of the camera, adjusting the location of one or more lenses ofthe camera, adjusting the location of one or more image sensors of thecamera, modifying the exposure time for images, changing the imageresolution captured or recorded by the camera, or performingpost-processing of images captured or recorded by the camera to correctfor defects/irregularities (e.g., by estimating parameters such as focallength or radial distortion to correct projections from a scene to acaptured or recorded image or to un-distort a captured or recordedimage).

A variety of cameras may benefit from calibration using the calibrationtarget. For example, the cameras calibrated may be components ofautonomous vehicles used for navigation or object recognition. Further,the cameras calibrated may have a variety of different features (e.g.,varying focal lengths, different lens types, different image sensors,etc.). The calibration target may be used to calibrate digital cameras(i.e., cameras that store images electrically or magnetically as aseries of bits), such as digital cameras having charge-coupled devices(CCDs) or complementary metal-oxide-semiconductor (CMOS) image sensors,or to calibrate film cameras (i.e., cameras that store images chemicallyon a strip of photographic film).

The first panel 102 may include a series of fiducial markers 110arranged thereon. It is understood that, in order to avoid clutter inthe illustration, only one of the fiducial markers 110 is labeled inFIG. 1A. The fiducial markers 110 are further described below withreference to FIG. 1C. Briefly, however, each fiducial marker 110 mayinclude a region used to identify that a fiducial marker has been found(e.g., during image analysis as part of calibration), a region toidentify which fiducial marker 110 it is, a region to pinpoint a centerof the fiducial marker 110, and a human-readable label of the fiducialmarker 110. A subset of the fiducial markers 110 may be used to identifythe first panel 102 as the first panel 102. These fiducial markers 110may be referred to as first-panel-identification fiducial markers 108.For example, the fiducial markers 110 nearest to the corners of thefirst panel 102 may indicate the identity of the first panel 102.

Another subset of the fiducial markers 110 (e.g., all of the fiducialmarkers 110 other than the first-panel-identification fiducial markers108) may be in a first arrangement of fiducial markers 110. The firstarrangement of fiducial markers 110 may include multiple patterns offiducial markers 110. For example, the first arrangement of fiducialmarkers 110 may include at least a first pattern of fiducial markers 110and a second pattern of fiducial markers 110. The second pattern offiducial markers 110 may be a scaled version of the first pattern offiducial markers 110. “Patterns” of fiducial markers 110 may be sets oflocations on the first panel 102 where fiducial markers 110 arepositioned. In various embodiments, there may be various patterns on thefirst panel 102 (described further with reference to FIGS. 6A-6E).Further, a “scaled version” may include a scaling of the locations ofthe fiducial markers (e.g., within a pattern) only, rather than ascaling of both the locations of the fiducial markers and the size ofthe fiducial markers. In alternate embodiments, however, both thescaling of both the locations of the fiducial markers and the size ofthe fiducial markers may be employed.

Illustrated in FIG. 1B is a second panel 104 of a calibration target.The second panel 104 may be a portion of the same calibration target asthe first panel 102. Similar to the first panel 102, the second panel104 may include a series of fiducial markers 110 arranged thereon.Again, only one of the fiducial markers 110 is labeled to avoidcluttering the figure. Like the first panel 102, a subset of thefiducial markers 110 may be used as second-panel-identification fiducialmarkers 108. For example, the fiducial markers 110 nearest to thecorners of the second panel 104 may indicate the identity of the secondpanel 104.

Similarly another subset of the fiducial markers 110 (e.g., all of thefiducial markers 110 other than the second-panel-identification fiducialmarkers 108) may be in a second arrangement of fiducial markers 110. Thesecond arrangement of fiducial markers 110 may include multiple patternsof fiducial markers 110. For example, the second arrangement of fiducialmarkers 110 may include at least a third pattern of fiducial markers anda fourth pattern of fiducial markers 110. The fourth pattern of fiducialmarkers 110 may be a scaled version of the third pattern of fiducialmarkers 110. In various embodiments, there may be various patterns onthe second panel 104. This will be described further with reference toFIGS. 6A-6E.

As illustrated in FIG. 1B, the fiducial markers 110 in the secondarrangement of the second panel 104 may be rotations of the fiducialmarkers 110 in the first arrangement of the first panel 102. In theembodiment illustrated, the fiducial markers 110 in the secondarrangement are 180° rotations of the fiducial markers 110 that are incorresponding locations in the first arrangement. In alternateembodiments, alternate rotations are also possible (e.g., 15°, 30°, 45°,60°, 75°, 90°, 105°, 120°, 135°, 150°, 165°, 195°, 210°, 225°, 240°,255°, 270°, 285°, 300°, 315°, 330°, 345°, or any values there between).In still other embodiments, only some of the fiducial markers 110 in thesecond arrangement may be rotated with respect to the fiducial markers110 in the first arrangement (e.g., only those fiducial markers 110corresponding to a particular subset of patterns of the firstarrangement).

The rotation of the fiducial markers 110 in the second arrangement withrespect to the fiducial markers 110 in the first arrangement 110 mayprovide features that distinguish the first panel 102 from the secondpanel 104. These additional features may be used in addition to thepanel-identification fiducial markers 108 to identify a given panel.Additionally or alternatively, all or a portion of one or more of thefiducial markers 110 in the first arrangement may be of a first colorand all or a portion of one or more of the fiducial markers 110 in thesecond arrangement may be of a second color. The first color and thesecond color may be different colors (e.g., blue and orange, yellow andred, brown and silver, light purple and dark purple, forest green andneon green, etc.), which may provide yet another feature that can beused to distinguish between the first panel 102 and the second panel104.

When designing the first panel 102 and the second panel 104 illustratedin FIGS. 1A and 1B, respectively, the fiducial markers 110 may bedesigned based on an anticipated location of the camera duringcalibration, the focal length of the camera, the zoom of the cameraduring calibration, and/or the resolution of the camera. The smaller thefiducial markers are with respect to the first panel 102 or the secondpanel 104, the more fiducial markers 110 can fit on the respectivepanel. Increasing the number of fiducial markers 110 that can fit on therespective panels can increase the number of calibration points (e.g.,positions in three-dimensional space whose corresponding two-dimensionallocation is used to calibrate an image), and ultimately increase theaccuracy of a calibration. However, the fiducial markers 110 need to beof at least a minimum resolvable size based on the camera given itszoom, resolution, focal length, distance from the calibration target,etc. Thus, the fiducial markers 110 on the first panel 102 and thesecond panel 104 may be sized such that they are of the smallest sizestill resolvable by the camera, such that the maximum number of fiducialmarkers 110 can be positioned on the first panel 102 and the secondpanel 104. Further, the fiducial markers 110 on the first panel 102 andthe second panel 104 may be spaced relative to one another such that themaximum number of fiducial markers 110 can be positioned on the firstpanel 102 and the second panel 104.

In the embodiments of the first panel 102 and the second panel 104 ofFIGS. 1A and 1B, there are six patterns of fiducial markers 110 in thefirst arrangement and six patterns of fiducial markers 110 in the secondarrangement. This is illustrated in FIG. 1D. As illustrated in FIG. 1D,the first arrangement of the first panel 102 and the second arrangementof the second panel 104 include each fiducial marker 110 labeled with a“1,” “2,” “3,” “4,” “5,” or “6.” The fiducial markers 110 labeled inFIG. 1D with a “0” represent the panel-identification fiducial markers108. The fiducial markers 110 labeled with a “1” are each part of thefirst pattern, the fiducial markers 110 labeled with a “2” are each partof the second pattern, the fiducial markers 110 labeled with a “3” areeach part of the third pattern, the fiducial markers 110 labeled with a“4” are each part of the fourth pattern, the fiducial markers 110labeled with a “5” are each part of the fifth pattern, and the fiducialmarkers 110 labeled with a “6” are each part of the sixth pattern of thefirst arrangement and the second arrangement of the first panel 102 andthe second panel 104, respectively. As such, there are 72 fiducialmarkers in the first arrangement and the second arrangement (e.g.,twelve fiducial markers in each of six patterns) and 76 fiducial markerstotal on the first panel 102 and the second panel 104 (e.g., twelvefiducial markers in each of six patterns, and four panel-identificationfiducial markers). In other embodiments, there may be more or fewerfiducial markers in each arrangement or on each panel. An increasednumber of fiducial markers may increase the resolution with which thecalibration of a camera can be performed. Alternatively, a reducednumber of fiducial markers may increase the speed at which thecalibration of a camera can be performed.

As illustrated in FIG. 1D, the fifth pattern and the sixth pattern arescaled versions (e.g., rectangles that have been scaled, potentiallyindependently, in both a horizontal direction and a vertical direction)of the first, second, third, and fourth patterns. In alternateembodiments, there may only be two patterns in the firstarrangement/second arrangement. In such embodiments, a second patternmay be a scaled version of the first pattern. In still otherembodiments, there may be any number of patterns (e.g., 3, 4, 5, 7, 8,9, 10, 15, 20, 50, or 100 patterns). Further, any number of patterns maybe scaled versions of any other patterns.

It is understood that the Arabic numerals used to label the fiducialmarkers 110 in FIG. 1D are not meant to necessarily represent the actualappearance of fiducial markers 110 in reductions to practice, but areinstead used to illustrate the various patterns of the first panel102/second panel 104.

FIG. 1C is an illustration of a fiducial marker 110 of a calibrationtarget. The fiducial marker 110 may be one of the fiducial markers 110of the first panel 102 or the second panel 104 illustrated in FIGS. 1Aand 1B, respectively. Illustrated on the left of FIG. 1C is an examplefiducial marker 110 as it may appear on an example embodiment of thefirst panel 102 (e.g., a first-panel-identification fiducial marker108). Illustrated on the right of FIG. 1C is a blank fiducial markerwhere different sections of the fiducial marker are labeled foridentification. As illustrated in FIG. 1C, the fiducial markers 110 ofthe first panel 102 and/or the second panel 104 may be circular inshape. Also as illustrated, fiducial markers may include crosshairs 112,an angular barcode section 114, a fiducial bounding section 116, and ahuman-readable label 118.

The fiducial marker 110 illustrated in FIG. 1C may be designed in such away as to accomplish multiple goals. First, the fiducial marker 110 maybe designed such that its center can be located (e.g., thetwo-dimensional center can be determined based on the fiducial marker).In addition, the fiducial marker 110 may be designed such that itsidentity can be determined among other fiducial markers on the samepanel or calibration target. When performing image analysis on thefiducial marker 110, a processor executing a set of calibrationinstructions may establish the outer bounds of the fiducial marker 110using the fiducial bounding section 116, determine which fiducial markeris being analyzed using the angular barcode section 114, and determinewhere the center of the fiducial marker 110 is using the crosshairs 112.

In alternate embodiments, the fiducial markers may have various othershapes (e.g., a triangular shape, a rectangular shape, a pentagonalshape, a hexagonal shape, a heptagonal shape, an octagonal shape, anonagonal shape, a decagonal shape, etc.). Further, within a singleembodiment, different fiducial markers may have different shapes. Forexample, the fiducial markers in the first pattern of the firstarrangement may be rectangular and the fiducial markers in the secondpattern of the first arrangement may be triangular. Multiple differentsubsets of fiducial markers on the same panel, arrangement, or patternmay have different shapes to identify different portions of therespective panel, arrangement, or pattern. Additionally oralternatively, fiducial markers in some embodiments may have specialoptical properties (e.g., a fiducial marker may be holographic).

The crosshairs 112 may be used to identify the center of the fiducialmarker 110 (e.g., where the center of the fiducial marker 110 islocated). The crosshairs 112 may alternatively be referred to as a“reticle,” in some embodiments. The location of a specific fiducialmarker (e.g., the horizontal and vertical coordinates within acalibration image) and the identity of the fiducial marker may both beused in calibrating a camera using a calibration target that includesone or more fiducial markers thereon. In some embodiments, thecrosshairs 112 may include the intersection of two dark regions and twolight regions, as illustrated in FIG. 1C. In alternate embodiments, thecrosshairs 112 may use a dot on the center of the fiducial marker, twoperpendicular lines crossing on the center of the fiducial marker, acircle whose center is the center of the fiducial marker, an arrow orline that points to or originates from the center of the fiducialmarker, or a chevron that points to the center of the fiducial marker.

The angular barcode section 114 may be used to identify a fiducialmarker within a calibration image. In the embodiment illustrated in FIG.1C, the angular barcode section 114 may be an annular sectionsurrounding the crosshairs 112. The angular barcode section 114 may bebroken into a series of bits (e.g., 4 bits, 8 bits, 10 bits, 16 bits, 24bits, 32 bits, 64 bits, 128 bits, 256 bits, etc.), each bit representedby a specific angular portion of the angular barcode section 114. Forexample, if the angular barcode section 114 is broken into 8 bits, thefirst bit may be represented by a portion of the angular barcode section114 that runs from about 0°-about 45° (progressing counterclockwisearound the angular barcode section 114) in the first quadrant of theangular barcode section 114. Similarly the second bit may run from about45°-about 90° in the first quadrant, the third bit may run from about90°-about 135° in the second quadrant, the fourth bit may run from about135°-about 180° in the second quadrant, the fifth bit may run from about180°-about 225° in the third quadrant, the sixth bit may run from about225°-about 270° in the third quadrant, the seventh bit may run fromabout 270°-about 315° in the fourth quadrant, and the eighth bit may runfrom about 315°-about 360° in the fourth quadrant. Each bit may befilled in, either in black (e.g., representing a 0) or in white (e.g.,representing a 1). Using this schema, various numbers can represented inbinary in the angular barcode section 114.

The numbers represented in the angular barcode section 114 may be usedto identify a given fiducial marker. As illustrated in FIGS. 1A and 1B,all of the fiducial markers within the first arrangement and the secondarrangement may have unique angular barcode sections; thus making eachfiducial marker in the first arrangement uniquely identifiable and eachfiducial marker in the second arrangement uniquely identifiable. In someembodiments, in addition to or alternate to the angular barcode section114, the location of a fiducial marker on a panel, the location of afiducial marker within an arrangement or pattern, the size of a fiducialmarker, the orientation of a fiducial marker, or the color of a fiducialmarker may be used to uniquely identify a fiducial marker. For example,in some embodiments, each fiducial marker may appear the same, but basedon their positions relative to the rest of the fiducial markers in anarrangement and their positions on a panel, each fiducial marker may beunique identified. Such may be the case in embodiments where thefiducial markers are squares in a checkerboard pattern, for instance.

Further, each of the angular barcode sections in the first arrangementand the second arrangement of some embodiments, including theembodiments illustrated in FIGS. 1A and 1B, may be rotationally unique.This means that, for a given angular barcode of a given fiducial marker,even if that fiducial marker and/or that angular barcode is rotated byany angle between 0° and 360°, it will not match another angular barcodebeing used. As an example, if the angular barcode 1-0-0-0-0-0-0-0-0-0(the first bit being the most significant bit, the last bit being theleast significant bit) were used in a set of angular barcodes thatincluded ten bits and was rotationally unique, the angular barcodes0-1-0-0-0-0-0-0-0-0, 0-0-1-0-0-0-0-0-0-0, 0-0-0-1-0-0-0-0-0-0,0-0-0-0-1-0-0-0-0-0, 0-0-0-0-0-1-0-0-0-0, 0-0-0-0-0-0-1-0-0-0,0-0-0-0-0-0-0-1-0-0, 0-0-0-0-0-0-0-0-1-0, and 0-0-0-0-0-0-0-0-0-1 couldnot be used because they rotationally overlap with 1-0-0-0-0-0-0-0-0-0.Thus, in order to preserve the rotationally unique set, some angularbarcodes will be skipped over. Therefore, there is not necessarily a1-to-1 correlation between possible angular barcodes and angularbarcodes actually employed in a given pattern, arrangement, panel, orcalibration target, especially in embodiments having rotationaluniqueness among angular barcodes.

In some embodiments, however, all angular barcodes may be used. In suchembodiments, the angular barcodes may be oriented on the panels in sucha way that, even though the fiducial markers would rotationally overlap,the orientation of each fiducial marker prevents actual angular barcodeoverlapping.

Further, in some embodiments, the angular barcodes may be used in acalibration method to determine the angular orientation of therespective fiducial marker. For example, the angular orientation of afiducial marker may be determined according to the calibration method toan accuracy of 360° divided by the number of bits used to encode theangular barcode.

In the examples shown in FIGS. 1A and 1B, the only fiducial markershaving the same angular barcode sections are the panel-identificationfiducial markers 108 in the corners of the first panel 102 and thesecond panel 104. This is so that the observation of any corner of thefirst panel 102 or the second panel 104 yields the same identity for thepanel being observed.

In alternate embodiments, rather than using bits, decimal numbersranging from one to ten could be represented at each location of theangular barcode section 114. For example, a different color could beused to represent each digit 0-9 (e.g., similar to resistorcolor-coding, black=0, brown=1, red=2, orange=3, yellow=4, green=5,blue=6, violet=7, grey=8, and white=9) and each section could representa factor of ten. Thus, if an angular barcode section was broken intoeight subsections, each represented with one of the above colors, therewould be 10⁸ possible combinations.

In alternate embodiments, where there are more or fewer than eight bits,the angular distribution of the angular barcode section 114 may bespaced differently (e.g., if there are 4 bits, rather than 8 bits, eachbit may occupy roughly 90° of the angular barcode section rather thanroughly 45° of the angular barcode section 114). Illustrated in FIGS.1A-1C, the angular barcode sections 114 of the fiducial markers 110 aresplit into ten bits, rather than eight bits. As such, each bit mayoccupy about 36° of the angular barcode section.

Still further, in some embodiments (e.g., the embodiments of FIGS. 1Aand 1B), certain angular barcodes may be reserved (e.g., 2, 3, 4, 5, 6,7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, etc.) for inclusionin panel-identification fiducial markers 108. As illustrated in FIGS. 1Aand 1B, the angular barcodes corresponding to “1” and to “2” may be usedas the angular barcodes of panel-identification fiducial markers 108 forthe first panel 102 and the second panel 104.

In some embodiments (e.g., embodiments that include a large number ofuniquely identifiable fiducial markers), the angular barcode section maybe replaced or supplemented by alternate forms of fiducial markeridentification. For example, rather than one ring corresponding to anangular barcode section, there could be multiple rings of angularbarcode sections. Additionally or alternatively, color could be used toidentify the fiducial markers. As indicated above, color could be usedto increase the areal density of the angular barcode sections (e.g.,rather than using only black/white, using multiple colors to representdecimal numbers rather than binary numbers). Alternatively, otherportions of the fiducial marker (e.g., the crosshairs, thehuman-readable label, or the fiducial bounding section) may be colored.For example, the crosshairs may be colored using paint or ink thatreflects wavelengths in the visible range (e.g., wavelengths from about400 nm to about 700 nm spaced in 1 nm increments) in order to addanother identifier that can add uniqueness to each fiducial marker. Instill other embodiments, a checkerboard pattern, rather than an angularbarcode may be used to identify a panel. For example, if a fiducialmarker were shaped as a rectangle or a square, one of the corners of thefiducial marker could be reserved for a checkerboard pattern (e.g., afour-by-four checkerboard where each position represents, using black orwhite, one of sixteen bits in the pattern). Such checkerboard patternsmay be similar to Quick-Response (QR) codes used for identification. Instill other embodiments, linear barcodes, rather than angular barcodes,may be used.

The fiducial bounding section 116 may identify the edge of the fiducialmarker/define the region occupied by the fiducial marker. The fiducialbounding section 116 may also be used during calibration to identifywhether or not a fiducial marker is actually present at a given locationin a calibration image. For example, an image analysis program executedby a processor to analyze a calibration image may determine that aspecific location of a calibration image appears to be crosshairs 112.However, in order to ensure that the section identified as crosshairs isactually crosshairs of a fiducial marker and not just noise or anotherartifact in the calibration image, the image analysis program executedby the processor may look for a fiducial bounding section surroundingthe crosshairs. If a fiducial bounding section is surrounded bycrosshairs, the chance of noise or other artifact obscuring thecalibration results may be reduced.

In some embodiments, rather than identifying the crosshairs 112 firstand then searching for a fiducial bounding section surrounding thecrosshairs, an image analysis program executed by a processor may searchfirst for the fiducial bounding section 116. Upon identifying thefiducial bounding section 116 (e.g., by identifying an ellipse in acalibration image), the image analysis program executed by the processormay look for the crosshairs 112 to determine a center of the respectivefiducial marker. Alternatively, the image analysis program executed bythe processor may determine a centerpoint of the ellipse that definesthe fiducial bounding section 116 rather than looking for the crosshairs112 in the calibration image.

The human-readable label 118 may be included in the fiducial markers 110in some embodiments such that a human (e.g., a calibration engineer) canidentify each of the fiducial markers 110, both on the panel/calibrationtarget and within a captured or recorded calibration image. Thehuman-readable label 118 may include one or more Arabic numerals, Romannumerals, letters, punctuation symbols, and/or other symbols or images,in various embodiments. In various embodiments, each of thehuman-readable labels 118 across a pattern, across an arrangement,across a panel, or across a calibration target may be unique (e.g.,similar to the angular barcode section 114 of the fiducial marker 110).Further, in some embodiments, the human-readable label 118 of thefiducial markers 110 may correspond to the angular barcode sections 114of the fiducial marker. For example, as illustrated in FIG. 1C, theArabic numeral “1” may be used as a human-readable label 118 for afiducial marker 110 that has the first angular barcode among a series ofangular barcodes in the angular barcode section 114 of the fiducialmarker 110. In some embodiments, there may be multiple human-readablelabels for each fiducial marker or no human-readable labels for eachfiducial marker. Further, in alternate embodiments, the human-readablelabel may be positioned differently than illustrated in FIG. 1C withrespect to the rest of the fiducial marker (e.g., the human-readablelabel may be in a lower-left region of the fiducial marker, rather thanin an upper-right region of the fiducial marker).

In some embodiments, portions of the calibration target may representportions of or entire advertisements. For example, if a panel is aportion of billboard (e.g., near a highway used for automotive transit),the panel could be dual-purposed (e.g., the panel could be used forcalibration of a camera equipped on an autonomous vehicle and toadvertise to riders within the autonomous vehicle). In such cases, eachfiducial marker may be an advertisement, or together, the firstarrangement of fiducial markers may make up an advertisement (e.g., thefirst pattern or the first arrangement may spell out a slogan).Additionally or alternatively, portions of each of the fiducial markersthemselves could be used for advertising. For example, a portion of thecrosshairs of one or more fiducial markers may include trade slogans,discounts, or other forms of advertising.

FIG. 1D is an additional illustration of the first panel 102 and thesecond panel 104 with fiducial markers thereon. FIG. 1D shows themultiple patterns of fiducial markers 110, as illustrated in FIGS. 1Aand 1B, in the first arrangement on the first panel 102 and the secondarrangement in the second panel 104. As illustrated, FIG. 1D also showsthe panel-identification fiducial markers 108 in the corners of thefirst panel 102/second panel 104. The numbering (0-6) in FIG. 1D ismeant to illustrate patterns made up of subsets of fiducial markers 110.The fiducial markers 110 having a “0” may be the panel-identificationfiducial markers 108. The fiducial markers 110 having a “1” may be in afirst pattern of the first arrangement of the first panel 102 or in athird pattern of the second arrangement of the second panel 104.Similarly, the fiducial markers 110 having a “2” may be in a secondpattern of the first arrangement of the first panel 102 or in a fourthpattern of the second arrangement of the second panel 104, and so on.

As illustrated, there may be six patterns of varying shapes and sizesper panel, plus a set of panel-identification fiducial markers 108.These patterns may be part of one or more arrangements (i.e., groups ofpatterns). It is understood that the naming convention “first pattern,”“second pattern,” “third pattern,” etc. is used only for identification,and that various embodiments may have various numbers of patternsidentified in various ways. For example, in one embodiment the “secondpattern” may be in the same locations as the “sixth pattern” in adifferent embodiment.

FIGS. 1E and 1F illustrate part of a calibration process using analternative calibration target 152. One way to calibrate a camera (e.g.,an image sensor with an associated lens) includes using the pinholecamera approximation model. In order to do this, a series of calibrationimages of a calibration target can be captured or recorded. Thecalibration target may have calibration markers (e.g., fiducials)thereon. The calibration markers may be spaced evenly or unevenly acrossthe calibration target. Either way, the location of the calibrationmarkers on the calibration target may be known a priori to thecalibration. Because the location of the calibration markers on thecalibration target is known, any deviation from those locations in theresulting captured or recorded calibration images can be identified.These identified deviations may then be corrected or accounted for whenusing the camera (i.e., the camera may be “calibrated”).

In order to determine a camera matrix associated with a camera that isto be calibrated, a series of calibration images of the calibrationtarget may be captured or recorded from different perspectives. Forexample, as illustrated in FIGS. 1E and 1F, a first calibration imagemay be captured or recorded from a first camera depth 162 (e.g., depthmeasured along a z-axis) relative to a calibration target 152, and thenthe camera may be moved (or the target may be moved) to a second cameradepth 164 relative to the calibration target 152 and a secondcalibration image captured or recorded. In this way, two differentcamera depths relative to the calibration target may be used (e.g., inthe determination of the camera matrix).

Illustrated in FIG. 1G is a portion of a calibration method using thefirst panel 102 illustrated in FIG. 1A. It is understood that theprinciple illustrated here equally applies if using the second panel 104illustrated in FIG. 1B. In FIG. 1G, a single calibration image of thefirst panel 102 may be recorded from a first calibration depth 172. Thefirst calibration depth 172 may be set such that the first panel 102 isa distance away from the camera equal to the focal distance (e.g., thedistance away from the camera at which an object is most in focus). Insome embodiments where the focal distance of the camera may be so largethat such a distance is not practical (e.g., when the focal distance is50 meters or more), the first calibration depth 172 may be set such thatthe first panel is closer to the camera than the focal distance of thecamera (e.g., based on the focal length of a lens of the camera and/orbased on a distance between the lens and an image sensor of the camera).In such embodiments however, in order to compensate for potentialeffects of the fiducial markers being out of focus in a calibrationimage, the first panel may be fabricated with larger fiducial markers(e.g., so they are detectable/distinguishable even when out of focus).The technique of designing a calibration target with larger fiducialmarkers to account for focal distance can be used with many embodimentsof the calibration target described herein (i.e., not only a singlepanel embodiment).

As illustrated in FIG. 1D, the first panel 102 may include a series ofpatterns. The patterns may be of various sizes and shapes. For example,a first pattern (e.g., the fiducial markers 110 labeled with a “1”)illustrated in FIG. 1D is a large rectangular pattern, while a secondpattern (e.g., the fiducial markers 110 labeled with a “5”) may be thesame shape as the first pattern, but may be a different size. In theembodiment of FIG. 1D, such a second pattern is a scaled version of thefirst pattern (e.g., the rectangular shape is the same, but theperimeter and area of the rectangle are both scaled down). Further, thesize of the fiducial markers 110 and the number of fiducial markers 110(e.g., twelve) are the same in each of the two patterns. In alternateembodiments, the size of the fiducial markers and the number of fiducialmarkers among patterns may be changed.

Because there are repeated patterns of fiducial markers 110 on the firstpanel 102 that have different scales, a calibration image captured orrecorded at the first calibration depth 172 can be used to mimic (i.e.,simulate) multiple calibration images captured or recorded at multiplecalibration depths. For example, a portion of the calibration image thatincludes the larger pattern on the first panel 102 (e.g., the firstpattern labeled with “1” in FIG. 1D) can be used as a calibration imagecaptured or recorded at the first calibration depth 172. In addition, aportion of the calibration image that includes the smaller pattern ofthe first panel 102 (e.g., the second pattern labeled with “5” in FIG.1D) can be used as if it were a calibration image captured or recordedat a second calibration depth 174. This eliminates the need to actuallycapture or record an additional calibration image from the secondcalibration depth 174. Such a phenomenon may have many benefits over thecalibration method illustrated in FIGS. 1E and 1F. For example, the timerequired for calibration may be reduced (both in terms of capturing orrecording calibration images and computation), the cost of thecalibration may be reduced, the amount of memory used for storingcalibration images may be reduced, and the physical space used for thecalibration may be reduced.

In some embodiments, only a portion of a calibration image may beanalyzed in order to perform calibration. For example, in embodimentswhere four panel-identification fiducial markers are at corners of apanel, such as in the panels illustrated in FIGS. 1A and 1B, the fourpanel-identification fiducial markers may bound all other fiducialmarkers (e.g., other fiducial markers on the panel are disposed inbetween the four panel-identification fiducial markers). As such, anyportion of a calibration image that lies outside of the fourpanel-identification fiducial markers may be disregarded during imageanalysis. Such a technique may save computational resources and reducecalibration time.

FIG. 2A illustrates, in a front view perspective (e.g., perspectiveparallel to the x-y plane), a calibration target used to calibrate oneor more cameras (e.g., one or more cameras used for navigation ofautonomous vehicles). The calibration target may include the first panel102, the second panel 104, and a connecting piece 202. As with above,only one of the fiducial markers 110 in FIG. 2A is labeled in order toavoid clutter on the figure. As illustrated, similar to as illustratedin FIGS. 1A and 1B, the first arrangement of fiducial markers 110 on thefirst panel 102 and the second arrangement of fiducial markers 110 thesecond panel 104 may be the same as one another. This may include one ormore patterns of fiducial markers 110 within the first arrangementmatching one or more patterns of fiducial markers 110 within the secondarrangement. Matching may include the patterns and/or the arrangementshaving similar or identical spatial arrangements, sizes, and/ororientations, in various embodiments.

Also as illustrated in FIGS. 1A and 1B, the fiducial markers 110themselves are similar in the first panel 102 and the second panel 104,but not identical. One primary difference is the panel-identificationfiducial markers 108 on the first panel 102 and the second panel 104. Asillustrated, the panel-identification fiducial markers 108 on the firstpanel 102 and the second panel 104 are in matching positions on therespective panels, but they are not the same fiducial markers. Notably,the human-readable labels are different (e.g., a “1” on the first panel102 and a “2” on the second panel 104). Further, the angular barcodesection 114 of the panel-identification fiducial markers 108 on the twopanels is different. Another difference between the first panel 102 andthe second panel 104 is that the fiducial markers in the secondarrangement on the second panel 104 have angular barcode sections 114that are 180° rotations of the corresponding angular barcode sections114 in the fiducial markers 110 in the first arrangement on the firstpanel 102. In some embodiments, however, the fiducial markers on thefirst panel 102 and the second panel 104 may be identical, with theexception of the panel-identification fiducial markers 108.

In even further embodiments, the panels of the calibration target maynot include panel-identification fiducial markers. Still further, insuch embodiments without panel-identification fiducial markers, thefiducial markers on the panels may be exactly duplicated (e.g., thefiducial markers on multiple panels may be in the same arrangements andpatterns on all panels). Because a calibration method used on thecalibration target may be based on known arrangements of fiducialmarkers on the panels, the location of each fiducial marker relative tothe rest of an arrangement is known (e.g., based on its angularbarcode). Further, if repeated copies of a given fiducial marker appearin a calibration image, a calibration method may account for this byrecognizing that multiple arrangements of fiducial markers are presentin the calibration image. If multiple arrangements of fiducial markersare present in the calibration image, and the location of each fiducialmarker within a given arrangement is known, errors regarding whicharrangement a given fiducial marker is in can be avoided.

In other embodiments, if repeated copies of a given fiducial markerappears in a calibration image, a calibration method may includedetermining that there are multiple arrangements present within thecalibration image. Further, the spatial relationship between themultiple arrangements may be determined (e.g., determining which patternis leftmost, rightmost, bottommost, topmost, second from the left,second from the right, second from the bottom, second from the top,etc.). Using the spatial relationship, a calibration method may includedetermining that one of the duplicate fiducial markers came from apattern whose location is known (e.g., if there are two copies of thesame fiducial marker within a calibration image, and one is located tothe left of the other, based on the spatial relationship betweenassociated arrangements, it may be determined that the fiducial markerlocated to the left is associated with the leftmost arrangement, whereinthe other fiducial marker is associated with the rightmost arrangement).Based on this determination, errors regarding which arrangement a givenfiducial marker is in can be avoided.

The calibration target illustrated in FIG. 2A may be used to calibratecameras at a fabrication facility. For example, an image sensor and/or alens may be calibrated using the calibration target after fabrication iscomplete (e.g., calibrated using a pinhole camera model). Additionallyor alternatively, the calibration target may be used to calibrate one ormore cameras upon assembly or installation. For example, when an imagesensor and a lens are together assembled within a housing to form adigital camera (e.g., in a laptop computer or a mobile phone), thedigital camera assembly may be calibrated. In still other embodiments,the calibration target may be used to calibrate a camera while in use.For example, if a camera is being used for navigation on an autonomousvehicle, the calibration target may be positioned at an exit of agarage, on a billboard, or on the road surface itself, such that whilethe autonomous vehicle is in motion calibration can be performed. Such acalibration may include a validation to a previously calibrated state(i.e., a “check-up” calibration). If the camera fails such a validation,the camera and/or the entire autonomous vehicle may be decommissionedpermanently or temporarily (e.g., until repair is performed). If thecamera is decommissioned, a redundant backup camera may be employed.

The first panel 102 and the second panel 104 may be fabricated out of avariety of materials. The fiducial markers 110 may be printed on thefirst panel 102 and the second panel 104. The connecting piece 202 maybe used to attach the first panel 102 to the second panel 104. Forexample, the connecting piece 202 may be welded to both the first panel102 and the second panel 104. As described further with respect to FIG.2B, the connected piece 202 may hold the first panel 102 and the secondpanel 104 at an angle relative to each other.

In alternate embodiments, the first panel 102 and the second panel 104may be directly connected to one another without using a connectingpiece. Further, in some embodiments, rather than using the entirecalibration target to calibrate a camera, only the first panel 102 orthe second panel 104 may be used to calibrate a camera.

FIG. 2B is a top-view illustration (e.g., perspective parallel to thex-z plane) of the calibration target in FIG. 2A. As illustrated, theconnecting piece 202 connects the first panel 102 and the second panel104 such that both are at 30° angles with respect to the connectingpiece 202. The dashed line is used to illustrate a line that iscollinear with the connecting piece 202. In alternate embodiments,different angles may be used (e.g., about 1°, about 2°, about 3°, about4°, about 5°, about 10°, about 15°, about 20°, about 25°, about 35°,about 40°, about 45°, about 50°, about 55°, about 60°, about 65°, about70°, about 75°, about 80°, about 85°, or about 90°). A 45° angle mayprovide the most variability between camera perspectives (i.e., thedifference in the appearance of the panels when moving relative to thecalibration target may be largest when the panels are at a 45° anglerelative to the connecting piece 202. In other embodiments, theconnecting piece itself may be bent to form an angle between the firstpanel and the second panel. Still further, in some embodiments, theangle between the first panel and the connecting piece may not be thesame as the angle between the second panel and the connecting piece(i.e., θ₁≠θ₂).

In some embodiments, the camera undergoing calibration may be positionedat a distance (e.g., in the z-direction) away from the calibrationtarget that is equal to the focal distance of the camera. For example,if the focal distance is 2 meters, calibration images of the calibrationtarget may be captured or recorded at a perspective from the calibrationtarget that is 2 meters away (e.g., in the z-direction). Additionally oralternatively, the camera undergoing calibration may be positioned at adistance (in the z-direction) relative to the calibration target, and/orthe calibration target may be sized, such that the calibration targetfills an entire field of view for the camera.

In addition, the calibration target may be a part of a calibrationsystem (e.g., used to calibrate a camera). The calibration system mayalso include a stage that is used to translate (e.g., stepwise) and/orrotate (e.g., stepwise) a camera relative to the calibration target(e.g., to capture or record multiple calibration images from multiplecamera perspectives). The stage may be controlled by a processor of acomputing device that executes a set of instructions to step the stagefrom one camera perspective to the next relative to the calibrationtarget. The camera may capture or record calibration images between orduring each step (e.g., as controlled by the processor).

FIG. 2C illustrates part of a calibration using an alternativecalibration target 250. As described with reference to FIGS. 1E and 1F,in order to determine a camera matrix associated with a camera, a seriesof calibration images of the calibration target may be captured orrecorded from different perspectives. For example, as illustrated inFIG. 2C, a first calibration image may be captured or recorded from afirst angular perspective 262 (e.g., about the y-axis) relative to acalibration target 250, and then the camera may be moved (or the targetmay be moved) to a second angular perspective 264 relative to thecalibration target 250 and a second calibration image captured orrecorded. In this way, two different camera angles relative to thecalibration target 250 may be used (e.g., in the determination of thecamera matrix).

Illustrated in FIG. 2D is a portion of a calibration method using thecalibration target illustrated in FIGS. 2A and 2B. In FIG. 2D, acalibration image of the calibration target may be captured or recordedfrom a single calibration position 272. As illustrated, the calibrationposition 272 may be aligned with the connecting piece 202 (e.g., thecalibration position 272 may be parallel to the connecting piece 202, interms of rotation, and the aperture of the camera may be roughlyhorizontally aligned, e.g., in the x-direction, with the connectingpiece 202). Further, the calibration position 202 may be at a depth(e.g., in the z-direction), d, relative to the calibration target. Insome embodiments, the depth may instead be defined as the distance(e.g., along the z-direction) from the camera to the horizontal centerof the panel, d′, or as the distance (e.g., along the z-direction) fromthe camera to the horizontal edge of the panel, d″. Additionally oralternatively, the calibration position 202 may be set such that thedepth d is equal to the focal distance of a lens on the camera beingcalibrated. As illustrated, the first panel 102 and the second panel 104may be disposed at an angle with respect to the connecting piece 202(e.g., a 30° angle).

Because the panels are at an angle with respect to one another, acalibration image captured or recorded at the calibration position 272can be used to mimic (e.g., simulate) multiple calibration imagescaptured or recorded at multiple calibration angles of a planarcalibration target (e.g., can be used to simulate two calibration imagestaken of the calibration target 250; one from the first angularperspective 262 and one from the second angular perspective 264). Insome embodiments, in order to use the calibration image captured orrecorded from the calibration position 272 to simulate calibrationimages from two angles (e.g., the first angular perspective 262 and thesecond angular perspective 264), a calibration image captured orrecorded from the calibration position 272 may be cropped. For example,the left half of the image may be used to mimic a calibration image ofthe calibration target 250 taken from the second angular perspective 264and the right half of the image may be used to mimic a calibration imageof the calibration target 250 taken from the first angular perspective262.

This may eliminate the need to actually capture or record calibrationimages from multiple angles. Such a phenomenon may have many benefitsover the multiple-location calibration method illustrated in FIG. 2C.For example, the time required for calibration may be reduced (both interms of capturing or recording calibration images and computation), thecost of the calibration may be reduced, the amount of memory used forstoring calibration images may be reduced, and the physical space usedfor the calibration may be reduced.

Because the first panel 102 and the second panel 104 may have scaledpatterns of fiducial markers 110, it is also understood that the featureof having a calibration target that enables multiple depths to besimulated (e.g., as illustrated in FIG. 1G) in a single calibrationimage may also be present in the embodiment of FIG. 2D. Thus, at leasttwo angles can be simulated and at least two depths can be simulatedusing the calibration target and calibration arrangement of in FIG. 2D.Hence, at minimum, one calibration image can simulate at least fourcalibration images, each taken from a different calibration location, ofa planar calibration target that does not employ fiducial markerscaling. In alternate embodiments, where additional scaling and/orangles of panels are used, even more calibration locations may besimulated. This is further described with reference to FIGS. 4A and 4B.

Illustrated in FIG. 3A is a third panel 302 of a calibration target. Thethird panel 302 may be a portion of the same calibration target as thefirst panel 102 and the second panel 104. Similar to the first panel 102and the second panel 104, the third panel 302 may include a series offiducial markers 110 arranged thereon. Again, only one of the fiducialmarkers 110 is labeled to avoid cluttering the figure. Like the firstpanel 102 and the second panel 104, a subset of the fiducial markers 110may be used as third-panel-identification fiducial markers 108. Forexample, the fiducial markers 110 nearest to the corners of the thirdpanel 302 may indicate the identity of the third panel 302. Asillustrated, the fiducial markers 110 on the third panel 302, with theexception of the third-panel-identification fiducial markers 108 in thecorners of the third panel 302, may match the fiducial markers 110 ofthe first panel 102 (as illustrated in FIG. 1A) and may be 180°rotations of the fiducial markers 110 of the second panel 104 (asillustrated in FIG. 1B).

As with the first panel 102 and the second panel 104, another subset ofthe fiducial markers 110 on the third panel 302 (e.g., all of thefiducial markers 110 other than the third-panel-identification fiducialmarkers 108) may be in a third arrangement of fiducial markers 110. Thethird arrangement of fiducial markers 110 may include multiple patternsof fiducial markers 110. For example, the third arrangement of fiducialmarkers 110 may include at least a fifth pattern of fiducial markers 110and a sixth pattern of fiducial markers 110. The sixth pattern offiducial markers 110 may be a scaled version of the fifth pattern offiducial markers 110. In various embodiments, there may be variouspatterns on the third panel 302. This will be described further withreference to FIGS. 6A-6E.

Illustrated in FIG. 3B is a fourth panel 304 of a calibration target.The fourth panel 304 may be a portion of the same calibration target asthe first panel 102, the second panel 104, and the third panel 302.Similar to the first panel 102, the second panel 104, and the thirdpanel 302, the fourth panel 304 may include a series of fiducial markers110 arranged thereon. Again, only one of the fiducial markers 110 islabeled to avoid cluttering the figure. Like the first panel 102, thesecond panel 104, and the third panel 302, a subset of the fiducialmarkers 110 may be used as fourth-panel-identification fiducial markers108. For example, the fiducial markers 110 nearest to the corners of thefourth panel 304 may indicate the identity of the fourth panel 304. Asillustrated, the fiducial markers 110 on the fourth panel 304, with theexception of the fourth-panel-identification fiducial markers 108 in thecorners of the fourth panel 304, may match the fiducial markers 110 ofthe second panel 104 (as illustrated in FIG. 1B) and may be 180°rotations of the fiducial markers 110 of the first panel 102 (asillustrated in FIG. 1A) and the third panel 302 (as illustrated in FIG.3A).

As with the first panel 102, the second panel 104, and the third panel302, another subset of the fiducial markers 110 on the fourth panel 304(e.g., all of the fiducial markers 110 other than thefourth-panel-identification fiducial markers 108) may be in a fourtharrangement of fiducial markers 110. The fourth arrangement of fiducialmarkers 110 may include multiple patterns of fiducial markers 110. Forexample, the fourth arrangement of fiducial markers 110 may include atleast a seventh pattern of fiducial markers 110 and an eighth pattern offiducial markers 110. The eighth pattern of fiducial markers 110 may bea scaled version of the seventh pattern of fiducial markers 110. Invarious embodiments, there may be various patterns on the fourth panel304. This will be described further with reference to FIGS. 6A-6E.

Similar to FIG. 2A, FIG. 4A illustrates, in a front-view perspective(e.g., perspective parallel to the x-y plane), a calibration target usedto calibrate one or more cameras (e.g., one or more cameras used fornavigation of autonomous vehicles). The calibration target may includethe first panel 102, the second panel 104, the third panel 302, thefourth panel 304, and multiple connecting pieces 402. As with above,only one of the fiducial markers 110 in FIG. 4A is labeled in order toavoid cluttering the figure. As illustrated, similar to as illustratedin FIG. 2A, the third arrangement of fiducial markers 110 on the thirdpanel 302 and the fourth arrangement of fiducial markers 110 on thesecond panel 104 may be the same as one another. This may include one ormore patterns of fiducial markers 110 within the third arrangementmatching one or more patterns of fiducial markers 110 within the fourtharrangement.

Similar to the calibration target illustrated in FIG. 2A, thecalibration target illustrated in FIG. 4A may be used to calibratecameras at a fabrication facility. Additionally or alternatively, thecalibration target may be used to calibrate one or more cameras uponassembly or installation.

FIG. 4B is a top-view illustration (e.g., perspective parallel to thex-z plane) of the calibration target in FIG. 4A. As illustrated, one ofthe connecting pieces 402 may connect the first panel 102 and the secondpanel 104 such that both are at 30° angles with respect to theconnecting piece 402 (similar to the calibration target illustrated inFIG. 2B). Additionally, one of the connecting pieces 402 may connect thethird panel 302 and the fourth panel 304 such that both are at 20°angles with respect to the connecting piece 402. Still further, anotherof the connecting pieces 402 may connect the first panel 102 and thesecond panel 104 to the third panel 302 and the fourth panel 304. Thedashed lines are used to illustrate lines that are collinear with atleast one the connecting pieces 404. In some embodiments, the dashedlines may be parallel to one another and/or parallel to the x-axis (asillustrated in FIG. 4B). In alternate embodiments, different angles maybe used (e.g., about 1°, about 2°, about 3°, about 4°, about 5°, about10°, about 15°, about 20°, about 25°, about 35°, about 40°, about 45°,about 50°, about 55°, about 60°, about 65°, about 70°, about 75°, about80°, about 85°, or about 90°). A 45° angle may provide the mostvariability between perspectives (i.e., the difference in the appearanceof the panels when moving relative to the calibration target may belargest when the panels are at a 45° angle relative to the connectingpiece 202). In some embodiments, the connecting pieces 404 maythemselves be angled to provide one or more angles between the fourpanels. Still further, in some embodiments, the angle between the firstpanel and the connecting piece may not be the same as the angle betweenthe second panel and the connecting piece (i.e., θ₁≠θ₂) and/or the anglebetween the third panel and the connecting piece may not be the same asthe angle between the fourth panel and the connecting piece (i.e.,Φ₁≠Φ₂). In some embodiments, θ₁=Φ₁, θ₂=Φ₂, θ₁=Φ₂, or θ₂=Φ₁.

In some embodiments, the middle connecting piece 402, which connects thetwo pairs of panels, may connect the pairs of panels in such a way thatthey are at an angle with one another. Said another way, the dashedlines illustrated in FIG. 4B, in some embodiments, may not be parallelto one another. Having this additional angle between the two pairs ofpanels may allow for additional simulated calibration images from asingle image (see description with respect to FIGS. 1E-1G and FIGS. 2Cand 2D).

In alternate embodiments, rather than a center connecting piececonnecting the pair of first and second panels with the pair of thirdand fourth panels, the two pairs of panels may be unconnected andindependently positioned relative to one another. This may allow for thetwo pairs of panels to be at a variety of angles with respect to oneanother. Further, such an angle between the panels may be adjustable bymoving one or both of the pairs of panels. Even further, in someembodiments, the two pairs of panels may be placed on opposite sides ofthe camera. In such embodiments, the camera may rotate to capture orrecord multiple calibration images so as to capture or record each ofthe two pairs of panels. Additionally or alternatively, the camera mayinclude a wide-angle lens or a fisheye lens to capture all or portionsof both pairs of panels from a single camera angle/position. In suchembodiments, the wide-angle lens or fisheye lens may be an inherentcomponent of the camera (e.g., rather than a lens added solely forcalibration).

In some embodiments where the two pairs of panels (i.e., two subsectionsof the same calibration target) are positioned on opposite sides of thecamera to be calibrated, the two pairs of panels may be differentdistances from the camera. During calibration, having panels atdifferent distances from the camera may bound the focal distancerelative to the camera (e.g., by including one set of panels located ata distance greater than the focal distance and one set of panels locatedat a distance less than the focal distance). In other embodiments, theremay be more than two pairs of panels placed at varying angles andlocations relative to the camera to be calibrated. In still otherembodiments, rather than pairs of panels, a single panel or three ormore panels may be used at one or more of the locations.

As illustrated in FIGS. 4C-4E, in some embodiments the two pairs ofpanels may be arranged in an inverted pyramid orientation relative tothe camera (e.g., where the base of the inverted pyramid is facing thecamera, i.e., where the point of the pyramid is facing away from thecamera). For example, one or both of the two pairs of panels may berotated above a horizontal axis (e.g., the x-axis in FIGS. 4C-4E) inaddition to a vertical axis (e.g., the z-axis in FIGS. 4C-4E). Such ahorizontal-axis rotation may correspond to leaning one or both of thepairs of panels. As in FIGS. 4C-4E, a first pair of panels (e.g., thefirst panel 102, the second panel 104, and the connecting piece 402) maybe joined to a second pair of panels (e.g., the third panel 302, thefourth panel 304, and the connecting piece 402) using one or moreconnecting pieces 452. FIG. 4C may illustrate a front-view of the twopairs of panels, FIG. 4D may illustrate a top-view (e.g., or a top-viewrotated 180° relative to the front-view) of the two pairs of panels, andFIG. 4E may illustrate a side-view of the two pairs of panels.

The inverted pyramid structure for a calibration target, as illustratedin FIGS. 4C-4E, may allow four different angles. For example, the firstpanel 102 and the second panel 104 are at 30° angles relative to theconnecting piece 402, the third panel 302 and the fourth panel 304 areat 20° angles relative to the connecting piece 402, the first pair ofthe first panel 102 and the second panel 104 are at a 10° angle, and thesecond pair of the third panel 302 and the fourth panel 304 are at a 15°angle. In other embodiments, other angles may be used. In still otherembodiments, eight unique angles, rather than four unique angles, may beincluded (e.g., if the first panel 102, the second panel 104, the thirdpanel 302, and the fourth panel 304 are all at different angles). Thefact that more than two angles are included in the calibration targetmay allow for additional calibration images to be simulated using asingle calibration image. In yet other embodiments, rather than multiplepanels at angles with respect to one another (e.g., in an accordionstyle as illustrated in FIG. 4A), a panel may be shaped hemispherically.This may allow for a continuum of angles on a single panel relative tothe camera. In some embodiments, a non-inverted (as opposed to aninverted) pyramid structure may be used for a calibration target. Instill yet other embodiments, a panel may be shaped as a semi-sphere or asemi-cylinder (e.g., a half-cylinder).

FIG. 5 illustrates a panel 502 of a calibration target. The panel 502may be similar to the first panel 102 illustrated in FIG. 1B. Asillustrated, the panel 502 has fiducial markers 110 thereon (e.g.,located in patterns and/or arrangements on the panel 502). As before,only one of the fiducial markers 110 is labeled to prevent the figurefrom being cluttered. The difference between the first panel 102illustrated in FIG. 1B and the panel 502 in FIG. 5 is that thepanel-identification fiducial marker 108 is in a center portion of thepanel 502 rather than at the corners. As such, there is only onepanel-identification fiducial marker 108 in FIG. 5 , rather thanmultiple. In alternate embodiments, there may be both apanel-identification fiducial marker in a center portion of the paneland panel-identification fiducial markers around corners or edges of thepanel. Further, across a single calibration target thepanel-identification fiducial markers may be located in differentregions on different panels. For example, one panel may have apanel-identification fiducial marker in a center portion and another mayhave panel-identification fiducial markers at corners of the panel.

FIG. 6A is an illustration of a panel 602 of a calibration target (e.g.,similar to the first panel 102 illustrated in FIG. 1A). The panel 602may have multiple patterns of fiducial markers thereon. In FIG. 6A, afirst pattern may include fiducial markers labeled with a “1” and asecond pattern may include fiducial markers labeled with a “2”. Asillustrated, the second pattern may be a scaled version of the firstpattern. Also as illustrated, the panel 602 may havepanel-identification fiducial markers (e.g., labeled using a “0”) atcorners of the panel 602. Unlike the first panel 102 illustrated in FIG.1A, the panel 602 may only include two rectangular patterns (as opposedto six). In various embodiments, various numbers of patterns may beincluded on each panel of the calibration target. Further, in someembodiments, different panels of the same calibration target may includediffering numbers of patterns. Additionally or alternatively, in variousembodiments, patterns on panels of calibration targets may havedifferent shapes (e.g., other than rectangular). For example, in someembodiments, the patterns may be circular, triangular, pentagonal,hexagonal, heptagonal, octagonal, nonagonal, or decagonal in shape.

FIG. 6B is an illustration of a panel 604 of a calibration target (e.g.,similar to the panel 602 illustrated in FIG. 6A). Unlike the panel 602illustrated in FIG. 6A, however, the panel 604 in FIG. 6B has a singlepanel-identification fiducial marker (e.g., labeled with a “0”) locatedin a center portion of the panel 604, rather than multiplepanel-identification fiducial markers at corners of the panel. Further,the first pattern of fiducial markers (e.g., labeled with “1”) and thesecond pattern of fiducial markers (e.g., labeled with “2”) are in aslightly different orientation/shape with respect to the panel 604 inFIG. 6B than are the first and second patterns of fiducial markers inthe panel 602 of FIG. 6A.

FIG. 6C is an illustration of a panel 606 of a calibration target (e.g.,similar to the panel 602 illustrated in FIG. 6A). Unlike the panel 602illustrated in FIG. 6A, however, the panel 606 in FIG. 6C is a circularpanel. Additionally, similar to the panel 604 illustrated in FIG. 6B,the panel 606 has a single panel-identification fiducial marker (e.g.,labeled with a “0”) located in a center portion of the panel 606.Further, the shape of the first pattern of fiducial markers (e.g.,labeled with “1”) and the second pattern of fiducial markers (e.g.,labeled with “2”) is circular. Circular patterns of fiducial markers maymatch the shape of the panel 606 and thus allow for an efficient use ofspace on the panel 606.

FIG. 6D is an illustration of a panel 608 of a calibration target (e.g.,similar to the panel 604 illustrated in FIG. 6B). Unlike the panel 604illustrated in FIG. 6B, however, the panel 608 in FIG. 6D has fiducialmarkers that are rectangular (e.g., square) in shape, rather thancircular. Rectangular fiducial markers may match the shape of the panel608 and thus allow for an efficient use of space on the panel 608 oralignment of fiducial markers in patterns on the panel 608.

FIG. 6E is an illustration of a panel 610 of a calibration target (e.g.,similar to the panel 604 illustrated in FIG. 6B). Unlike the panel 604illustrated in FIG. 6B, however, the panel 610 in FIG. 6E has fiducialmarkers that are star-shaped, rather than circular. Further, asillustrated in FIG. 6E, the star-shaped fiducial markers may coincidewith a first pattern of fiducial markers (e.g., labeled with “1”) and asecond pattern of fiducial markers (e.g., labeled with “0”) havingfiducial markers placed within the respective patterns at points of astar shape. In other words, the star-shaped pattern of the fiducialmarkers may be mirrored by the first and second patterns.

III. EXAMPLE PROCESSES

FIG. 7 is a flowchart illustration of a method 700, according to exampleembodiments. The method 700 described may include one or moreoperations, functions, or actions as illustrated by one or more of theblocks. Although the blocks are illustrated in a sequential order, theseblocks may in some instances be performed in parallel, or in a differentorder than those described herein. Also, the various blocks may becombined into fewer blocks, divided into additional blocks, or removedbased upon the desired implementation. Further additional blocksdescribing additional, non-essential steps may be included in somevariations of the methods contemplated herein.

In some embodiments, one or more of the blocks of FIG. 7 may beperformed by a computing device. The computing device may includecomputing components such as a non-volatile memory (e.g., a hard driveor a read-only memory (ROM)), a volatile memory (e.g., a random-accessmemory (RAM), such as dynamic random-access memory (DRAM) or staticrandom-access memory (SRAM)), a user-input device (e.g., a mouse or akeyboard), a display (e.g., an light-emitting diode (LED) display or aliquid crystal display (LCD)), and/or a network communication controller(e.g., a WIFI® controller, based on IEEE 802.11 standards, or anEthernet controller). The computing device, for example, may executeinstructions stored on a non-transitory, computer-readable medium (e.g.,a hard drive) to perform one or more of the operations described herein.

At block 702, the method 700 may include capturing or recording acalibration image of a target using a camera. The target may include afirst panel having a first arrangement of fiducial markers thereon. Eachof the fiducial markers in the first arrangement may be unique amongfiducial markers in the first arrangement. The target may also include asecond panel, disposed at a first angle relative to the first panel,having a second arrangement of fiducial markers thereon. Each of thefiducial markers in the second arrangement may be unique among fiducialmarkers in the second arrangement. The first arrangement of the fiducialmarkers may be the same as (i.e., match) the second arrangement offiducial markers.

In some embodiments, the first panel may further include one or morefirst-panel-identification fiducial markers that uniquely identify thefirst panel. Additionally or alternatively, the first arrangement mayinclude a first pattern of fiducial markers and a second pattern offiducial markers. Further, the second pattern of fiducial markers may bea scaled version of the first pattern of fiducial markers. Stillfurther, the second panel may include one or moresecond-panel-identification fiducial markers that uniquely identify thesecond panel. The second arrangement may additionally include a thirdpattern of fiducial markers and a fourth pattern of fiducial markers.The third pattern of fiducial markers may match the first pattern offiducial markers and the fourth pattern of fiducial markers may matchthe second pattern of fiducial markers.

At block 704, the method 700 may include determining locations andidentifications of one or more fiducial markers in the calibrationimage.

At block 706, the method 700 may include, based on the determinedlocations and identifications, calibrating the camera.

Calibrating the camera may include determining correlations between thedetermined locations and identifications of the one or more fiducialmarkers in the calibration image and locations and identifications ofthe one or more fiducial markers on the calibration target. Based onthese determined correlations, parameters of a camera matrix may beestimated using a pinhole camera model. Using the pinhole camera modelmay include determining a three-dimensional location of a pinholerepresenting an aperture of the camera using the determinedcorrelations. Additionally or alternatively, calibrating the camera mayinclude accounting for radial distortions or tangential distortions.

In one example embodiment, the method 700 may include splitting thecaptured or recorded calibration images into multiple simulatedcalibration images (e.g., one simulated calibration image for eachpattern within each panel). The multiple simulated calibration images ofmultiple patterns may then be used to represent multiple actualcalibration images captured or recorded of a single pattern (at astationary 3D location) from multiple perspectives (e.g., multipleangles and/or multiple depths). Using the multiple simulated calibrationimages, camera parameters (e.g., parameters of a camera matrixcorresponding to the pinhole camera model) may be optimized. Thisoptimization may correspond to a minimized reprojection error in mappingthe 3D location of each fiducial marker to the 2D location of thefiducial markers in the calibration images.

In some embodiments, the locations and identifications of the one ormore fiducial markers on the calibration target (i.e., “ground truth”)may be established based on one or more methods used to print fiducialmarkers on panels of the calibration target, methods used to arrangepanels of the calibration target relative to one another, or otherfabrication/assembly methods. Additionally or alternatively, thelocations and identifications of the one or more fiducial markers on thecalibration target may be established by another optical procedure. Forexample, using a light detection and ranging (LIDAR) system, preciselocations of each of the fiducial markers within the calibration targetmay be determined.

Determining a three-dimensional location of a pinhole corresponding tothe aperture of the camera using the determined correlations may be anNP-hard problem. As such, in some embodiments, once the location of thepinhole has been determined, additional calibrations may be performed todetermine whether a camera matrix associated with the calibrated camerais still accurate to within a given degree (e.g., after a predefinedamount of time has elapsed during which the camera may have becomedetuned). Determining whether the camera matrix is accurate to within agiven degree may require fewer calculations, as such a comparison maynot be an NP-hard problem (whereas the original determination of thecamera matrix may be).

Further, calibrating the camera using the captured or recordedcalibration images may include determining an angle, relative to theimage sensor in the camera, where each pixel in the camera is facing.Determining an angle for each pixel may include generating a lookuptable of angles for each pixel. Alternatively, determining an angle foreach pixel may include generating a parametrization that describes theangles (e.g., a parameterization based on two, three, four, five, six,seven, eight, nine, ten, etc. variables). Generating a parametrizationmay be less computationally intensive than generating a lookup table.

In some embodiments, the method 700 may also include rotating the camerarelative to the target and capturing or recording an additionalcalibration image. Rotating the camera relative to the target mayinclude rotating the camera about a pitch axis (e.g., about an x-axis),about a roll axis (e.g., about a z-axis), about a yaw axis (e.g., abouta y-axis), or about a superposition of axes selected from among thepitch axis, roll axis, and yaw axis. Rotating the camera may allow forone or more additional calibration images to be captured or recordedfrom different perspectives relative to the target. Additionally oralternatively, rotating the camera may allow for additional calibrationimages to be captured or recorded such that the entirety of thecalibration target is captured or recorded among the set of calibrationimages. For example, if the field of view of the camera undergoingcalibration is narrower (in one or more dimensions) than the calibrationtarget based on the position of the camera relative to the calibrationtarget, multiple calibration images may be captured or recorded of thecalibration target such that the entirety of the calibration target iscaptured or recorded. In one embodiment, this may include capturing orrecording a calibration image of a first panel of a calibration target,rotating (or translating) the camera such that it faces a second panelof the calibration target, capturing or recording a calibration image ofthe second panel of the calibration target, and then performing acalibration of the camera using both the calibration image of the firstpanel and the calibration image of the second panel. In alternateembodiments, a single calibration image or multiple calibration imagesof the calibration target may be captured or recorded withouttranslating or rotating the target.

In some embodiments, the method 700 may also include translating thecamera relative to the calibration target and capturing or recording anadditional calibration image. Translating the camera relative to thecalibration target may include translating the camera in a horizontaldirection parallel to the calibration target (e.g., x-direction), ahorizontal direction normal to the calibration target (e.g.,z-direction), a vertical direction parallel to the calibration target(e.g., y-direction), or a superposition of directions selected fromamong the horizontal-parallel, horizontal-normal, and vertical-paralleldirection. Translating the camera may allow for one or more additionalcalibration images to be captured or recorded from differentperspectives relative to the target. A calibration of the camera maythen be performed using both the calibration image captured or recordedbefore translating the camera and the one or more additional calibrationimages captured or recorded from different perspectives aftertranslating the camera.

The method 700 may further include cropping the calibration image intotwo or more calibration sub-images (e.g., at least one sub-image of thefirst panel and at least one sub-image of the second panel). In thisway, individual panels of the calibration target can be isolated andanalyzed individually. Such an individual analysis of each calibrationsub-image may take into account the fact that the first panel and thesecond panel may be at different angles with respect to the cameracapturing or recording them (e.g., because the two different panels weresimulating different camera positions relative to the calibrationtarget).

In some embodiments, multiple calibration images may be captured orrecorded using the calibration target. For example, a computing devicemay be connected to a stage on which the camera undergoing calibrationis mounted (e.g., as part of a calibration system that includes thestage and the calibration target). The computing device may beconfigured to translate and rotate the stage relative to the calibrationtarget. In such embodiments, the computing device may translate thecamera relative to the calibration target stepwise. Additionally oralternatively, the computing device may rotate the camera relative tothe calibration target stepwise. As the camera is translated or rotatedin a stepwise fashion, calibration images may be captured or recorded ofthe calibration target from different perspectives of the camera. Eachof the calibration images may be used as additional data forcalibration. For example, a calibration target that has two panels at agiven angle may simulate two angles in a single calibration image, butif the camera rotates stepwise and captures or records additionalcalibration images, two additional calibration angles may be obtainedfor each additional calibration image captured or recorded.

IV. CONCLUSION

The present disclosure is not to be limited in terms of the particularembodiments described in this application, which are intended asillustrations of various aspects. Many modifications and variations canbe made without departing from its scope, as will be apparent to thoseskilled in the art. Functionally equivalent methods and apparatuseswithin the scope of the disclosure, in addition to those describedherein, will be apparent to those skilled in the art from the foregoingdescriptions. Such modifications and variations are intended to fallwithin the scope of the appended claims.

The above detailed description describes various features and operationsof the disclosed systems, devices, and methods with reference to theaccompanying figures. The example embodiments described herein and inthe figures are not meant to be limiting. Other embodiments can beutilized, and other changes can be made, without departing from thescope of the subject matter presented herein. It will be readilyunderstood that the aspects of the present disclosure, as generallydescribed herein, and illustrated in the figures, can be arranged,substituted, combined, separated, and designed in a wide variety ofdifferent configurations.

With respect to any or all of the message flow diagrams, scenarios, andflow charts in the figures and as discussed herein, each step, block,operation, and/or communication can represent a processing ofinformation and/or a transmission of information in accordance withexample embodiments. Alternative embodiments are included within thescope of these example embodiments. In these alternative embodiments,for example, operations described as steps, blocks, transmissions,communications, requests, responses, and/or messages can be executed outof order from that shown or discussed, including substantiallyconcurrently or in reverse order, depending on the functionalityinvolved. Further, more or fewer blocks and/or operations can be usedwith any of the message flow diagrams, scenarios, and flow chartsdiscussed herein, and these message flow diagrams, scenarios, and flowcharts can be combined with one another, in part or in whole.

A step, block, or operation that represents a processing of informationcan correspond to circuitry that can be configured to perform thespecific logical functions of a herein-described method or technique.Alternatively or additionally, a step or block that represents aprocessing of information can correspond to a module, a segment, or aportion of program code (including related data). The program code caninclude one or more instructions executable by a processor forimplementing specific logical operations or actions in the method ortechnique. The program code and/or related data can be stored on anytype of computer-readable medium such as a storage device including RAM,a disk drive, a solid state drive, or another storage medium.

The computer-readable medium can also include non-transitorycomputer-readable media such as computer-readable media that store datafor short periods of time like register memory and processor cache. Thecomputer-readable media can further include non-transitorycomputer-readable media that store program code and/or data for longerperiods of time. Thus, the computer-readable media may include secondaryor persistent long term storage, like ROM, optical or magnetic disks,solid state drives, compact-disc read only memory (CD-ROM), for example.The computer-readable media can also be any other volatile ornon-volatile storage systems. A computer-readable medium can beconsidered a computer-readable storage medium, for example, or atangible storage device.

Moreover, a step, block, or operation that represents one or moreinformation transmissions can correspond to information transmissionsbetween software and/or hardware modules in the same physical device.However, other information transmissions can be between software modulesand/or hardware modules in different physical devices.

The particular arrangements shown in the figures should not be viewed aslimiting. It should be understood that other embodiments can includemore or less of each element shown in a given figure. Further, some ofthe illustrated elements can be combined or omitted. Yet further, anexample embodiment can include elements that are not illustrated in thefigures.

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purpose ofillustration and are not intended to be limiting, with the true scopebeing indicated by the following claims.

Embodiments of the present disclosure may thus relate to one of theenumerated example embodiments (EEEs) listed below.

EEE 1 is a target used for calibration, comprising:

-   -   a first panel having a first arrangement of fiducial markers        thereon,    -   wherein each of the fiducial markers in the first arrangement is        uniquely identifiable among fiducial markers in the first        arrangement; and    -   a second panel, disposed at a first angle relative to the first        panel, having a second arrangement of fiducial markers thereon,    -   wherein each of the fiducial markers in the second arrangement        is uniquely identifiable among fiducial markers in the second        arrangement, and    -   wherein the first arrangement of fiducial markers matches the        second arrangement of fiducial markers.

EEE 2 is the target of EEE 1,

-   -   wherein the first panel further comprises one or more        first-panel-identification fiducial markers that uniquely        identify the first panel, and    -   wherein the second panel further comprises one or more        second-panel-identification fiducial markers that uniquely        identify the second panel.

EEE 3 is the target of EEE 2,

-   -   wherein the one or more first-panel-identification fiducial        markers are disposed in one or more corners of the first panel,        and    -   wherein the one or more second-panel-identification fiducial        markers are disposed in one or more corners of the second panel.

EEE 4 is the target of any of EEEs 1-3, wherein the fiducial markers inthe first arrangement and the second arrangement are circular in shape.

EEE 5 is the target of any of EEEs 1-4, wherein the fiducial markers inthe first arrangement and the second arrangement comprise crosshairsthat identify a center of each respective fiducial marker.

EEE 6 is the target of any of EEEs 1-5, wherein the fiducial markers inthe first arrangement and the second arrangement are each uniquelylabeled by respective angular barcodes.

EEE 7 is the target of EEE 6, wherein the angular barcodes comprise aten-bit encoding scheme where each bit is represented by a light or darksection representing a 1 or a 0.

EEE 8 is the target of EEE 6, wherein the angular barcodes arerotationally unique such that, even if rotated by any angle, the angularbarcodes will not match one another.

EEE 9 is the target of any of EEEs 1-8, further comprising:

-   -   a third panel having a third arrangement of fiducial markers        thereon,    -   wherein each of the fiducial markers in the third arrangement is        uniquely identifiable among fiducial markers in the third        arrangement, and    -   wherein the third panel further comprises one or more        third-panel-identification fiducial markers that uniquely        identify the third panel; and    -   a fourth panel, disposed at a second angle relative to the third        panel, having a fourth arrangement of fiducial markers thereon,    -   wherein each of the fiducial markers in the fourth arrangement        is uniquely identifiable among fiducial markers in the fourth        arrangement,    -   wherein the fourth panel further comprises one or more        fourth-panel-identification fiducial markers that uniquely        identify the fourth panel, and    -   wherein the second angle is not equal to the first angle.

EEE 10 is the target of EEE 9,

-   -   wherein the third panel and the fourth panel are disposed at a        third angle relative to the first panel and the second panel,        and    -   wherein the third angle is about a different axis than the first        angle and the second angle.

EEE 11 is the target of any of EEEs 1-10,

-   -   wherein the first arrangement comprises a first pattern of        fiducial markers and a second pattern of fiducial markers,    -   wherein the second arrangement comprises a third pattern of        fiducial markers and a fourth pattern of fiducial markers,    -   wherein the second pattern of fiducial markers is a scaled        version of the first pattern of fiducial markers, and    -   wherein the third pattern of fiducial markers matches the first        pattern of fiducial markers and the fourth pattern of fiducial        markers matches the second pattern of fiducial markers.

EEE 12 is the target of any of EEEs 1-11, wherein the fiducial markersin the second arrangement pattern are 180-degree rotations of thefiducial markers in the first arrangement at a corresponding location.

EEE 13 is the target of any of EEEs 1-12,

-   -   wherein the fiducial markers in the first arrangement comprise a        first color and the fiducial markers in the second arrangement        comprise a second color, and    -   wherein the first color and the second color are different        colors.

EEE 14 is a method, comprising:

-   -   recording a calibration image of a target using a camera,        wherein the target comprises:        -   a first panel having a first arrangement of fiducial markers            thereon,        -   wherein each of the fiducial markers in the first            arrangement is uniquely identifiable among fiducial markers            in the first arrangement; and        -   a second panel, disposed at a first angle relative to the            first panel, having a second arrangement of fiducial markers            thereon,        -   wherein each of the fiducial markers in the second            arrangement is uniquely identifiable among fiducial markers            in the second arrangement, and        -   wherein the first arrangement of fiducial markers matches            the second arrangement of fiducial markers; and    -   determining locations and identifications of one or more        fiducial markers in the calibration image; and    -   based on the determined locations and identifications,        calibrating the camera.

EEE 15 is the method of EEE 14, wherein calibrating the camera comprisesusing correlations between locations of the fiducial markers on thefirst panel and locations of the fiducial markers on the second panel toestimate parameters of a camera matrix using a pinhole camera model.

EEE 16 is the method of EEEs 14 or 15,

-   -   wherein the first arrangement comprises a first pattern of        fiducial markers and a second pattern of fiducial markers, and    -   wherein the second pattern of fiducial markers is a scaled        version of the first pattern of fiducial markers.

EEE 17 is the method of any of EEEs 14-16, further comprising:

-   -   rotating the camera relative to the target and recording an        additional calibration image; and    -   determining locations and identifications of one or more        fiducial markers in the additional calibration image.

EEE 18 is the method of EEE 17, wherein rotating the camera relative tothe target comprises rotating the camera about at least one of a pitchaxis of the camera or a roll axis of the camera.

EEE 19 is the method of any of EEEs 14-18, further comprising:

-   -   moving the camera and recording an additional calibration image;        and    -   determining locations and identifications of one or more        fiducial markers in the additional calibration image.

EEE 20 is a system used for calibrating a camera, comprising:

-   -   a target, comprising:        -   a first panel having a first arrangement of fiducial markers            thereon,        -   wherein each of the fiducial markers in the first            arrangement is uniquely identifiable among fiducial markers            in the first arrangement; and        -   a second panel, disposed at a first angle relative to the            first panel, having a second arrangement of fiducial markers            thereon,        -   wherein each of the fiducial markers in the second            arrangement is uniquely identifiable among fiducial markers            in the second arrangement, and        -   wherein the first arrangement of fiducial markers matches            the second arrangement of fiducial markers; and    -   a stage configured to translate or rotate the camera with        respect to the target.

EEE 21 is a target used for calibration, comprising:

-   -   a first pattern of fiducial markers; and    -   a second pattern of fiducial markers,    -   wherein the first pattern of fiducial markers is a scaled        version of the second pattern of fiducial markers, such that a        calibration image captured of the target simulates multiple        images of a single pattern captured at multiple calibration        perspectives.

EEE 22 is the target of EEE 21, wherein the fiducial markers in thefirst pattern and the fiducial markers in the second pattern arecircular in shape.

EEE 23 is the target of EEEs 21 or 22, wherein the fiducial markers inthe first pattern and the fiducial markers in the second patterncomprise crosshairs that identify a center of each respective fiducialmarker.

EEE 24 is the target of any of EEEs 21-23, wherein each of the fiducialmarkers in the first pattern and each of the fiducial markers in thesecond pattern are uniquely identifiable relative to other fiducialmarkers of the target.

EEE 25 is the target of EEE 24, wherein the fiducial markers in thefirst pattern and the fiducial markers in the second pattern are eachuniquely labeled by respective angular barcodes.

EEE 26 is the target of EEE 25, wherein the angular barcodes comprise aten-bit encoding scheme where each bit is represented by a light or darksection representing a 1 or a 0.

EEE 27 is the target of EEE 25, wherein the angular barcodes arerotationally unique such that, even if rotated by any angle, the angularbarcodes will not match one another.

EEE 28 is a method, comprising:

-   -   recording a calibration image of a target using a camera,        wherein the target comprises:        -   a first panel having a first arrangement of fiducial markers            thereon,        -   wherein each of the fiducial markers in the first            arrangement is uniquely identifiable among fiducial markers            in the first arrangement; and        -   a second panel, disposed at a first angle relative to the            first panel, having a second arrangement of fiducial markers            thereon,        -   wherein each of the fiducial markers in the second            arrangement is uniquely identifiable among fiducial markers            in the second arrangement, and        -   wherein the first arrangement of fiducial markers matches            the second arrangement of fiducial markers; and    -   determining locations and identifications of one or more        fiducial markers in the calibration image; and    -   based on the determined locations and identifications,        calibrating the camera.

EEE 29 is the method of EEE 28, wherein the fiducial markers in thesecond arrangement are 180-degree rotations of the fiducial markers inthe first arrangement at a corresponding location.

EEE 30 is the method of EEEs 28 or 29,

-   -   wherein the fiducial markers in the first arrangement comprise a        first color and the fiducial markers in the second arrangement        comprise a second color, and    -   wherein the first color and the second color are different        colors.

EEE 31 is the method of any of EEEs 28-30,

-   -   wherein the first panel comprises one or more        first-panel-identification fiducial markers that uniquely        identify the first panel, and    -   wherein the second panel comprises one or more        second-panel-identification fiducial markers.

EEE 32 is the method of EEE 31,

-   -   wherein the one or more first-panel-identification fiducial        markers are disposed in one or more corners of the first panel,        and    -   wherein the one or more second-panel-identification fiducial        markers are disposed in one or more corners of the second panel.

EEE 33 is the method of any of EEEs 28-32, wherein the target furthercomprises:

-   -   a third panel having a third arrangement of fiducial markers        thereon,    -   wherein each of the fiducial markers in the third arrangement is        uniquely identifiable among fiducial markers in the third        arrangement; and    -   a fourth panel, disposed at a second angle relative to the third        panel, having a fourth arrangement of fiducial markers thereon,    -   wherein each of the fiducial markers in the fourth arrangement        is uniquely identifiable among fiducial markers in the fourth        arrangement,    -   wherein the third arrangement of fiducial markers and the fourth        arrangement of fiducial markers each match the first arrangement        of fiducial markers and the second arrangement of fiducial        markers, and    -   wherein the second angle is not equal to the first angle.

EEE 34 is the method of EEE 33,

-   -   wherein the third panel and the fourth panel are disposed at a        third angle relative to the first panel and the second panel,        and    -   wherein the third angle is about a different axis than the first        angle and the second angle.

EEE 35 is the method of any of EEEs 28-34, wherein calibrating thecamera comprises using correlations between locations of the fiducialmarkers on the first panel and locations of the fiducial markers on thesecond panel to estimate parameters of a camera matrix using a pinholecamera model.

EEE 36 is the method of any of EEEs 28-35,

-   -   wherein the first arrangement comprises a first pattern of        fiducial markers and a second pattern of fiducial markers, and    -   wherein the second pattern of fiducial markers is a scaled        version of the first pattern of fiducial markers.

EEE 37 is the method of any of EEEs 28-36,

-   -   rotating the camera relative to the target and recording an        additional calibration image; and    -   determining locations and identifications of one or more        fiducial markers in the additional calibration image.

EEE 38 is the method of EEE 37, wherein rotating the camera relative tothe target comprises rotating the camera about at least one of a pitchaxis of the camera or a roll axis of the camera.

EEE 39 is the method of any of EEEs 28-38, further comprising:

-   -   moving the camera and recording an additional calibration image;        and    -   determining locations and identifications of one or more        fiducial markers in the additional calibration image.

EEE 40 is a system used for calibrating a camera, comprising:

-   -   a target, comprising:        -   a first pattern of fiducial markers; and        -   a second pattern of fiducial markers,        -   wherein the first pattern of fiducial markers is a scaled            version of the second pattern of fiducial markers, such that            a calibration image captured of the target simulates            multiple images of a single pattern captured at multiple            calibration perspectives; and    -   a stage configured to translate or rotate the camera with        respect to the target.

What is claimed:
 1. A target used for calibration, comprising: a panelhaving a first plurality of fiducial markers arranged on the panel in afirst pattern and a second plurality of fiducial markers arranged on thepanel in a second pattern, wherein the first pattern is a scaled versionof the second pattern, such that a calibration image captured of thetarget simulates multiple images of a single pattern captured atmultiple calibration perspectives, and wherein fiducial markers of thefirst plurality of fiducial markers have a different shape than fiducialmarkers of the second plurality of fiducial markers, wherein at leastone fiducial marker of at least a portion of the first and secondpluralities of fiducial markers comprises a plurality of distinctregions, and wherein the plurality of distinct regions comprises atleast two of: a first region having a crosshair that indicates a centerof the fiducial marker, a second region having an angular barcode thatuniquely identifies the fiducial marker within a calibration image takenof the fiducial marker, a third region that identifies a boundary of thefiducial marker, or a fourth region having a human-readable label thatuniquely identifies the fiducial marker within the calibration imagetaken of the fiducial marker.
 2. The target of claim 1, wherein thefirst pattern is a first rectangular pattern, wherein the second patternis a second rectangular pattern, and wherein the first rectangularpattern is scaled such that the first rectangular pattern has largerhorizontal and vertical dimensions than the second rectangular pattern.3. The target of claim 1, wherein the first region is circular in shapeand approximately centered at a center of the fiducial marker, whereinthe second region is annular in shape and is positioned adjacent to aperimeter of the first region, and wherein the third region is annularin shape and is positioned adjacent to a perimeter of the second region.4. The target of claim 1, wherein the panel is the only panel the targetcomprises.
 5. The target of claim 1, wherein the panel further comprisesa panel-identification fiducial marker that uniquely identifies thetarget.
 6. The target of claim 5, wherein the panel-identificationfiducial marker is disposed in a corner of the panel.
 7. A method,comprising: recording a calibration image of a target using a camera,wherein the target comprises a single panel having a first plurality offiducial markers arranged on the single panel in a first pattern and asecond plurality of fiducial markers arranged on the single panel in asecond pattern, wherein each of the fiducial markers in the first andsecond pluralities of fiducial markers is uniquely identifiable amongthe first and second pluralities of fiducial markers, and whereinfiducial markers of the first plurality of fiducial markers have adifferent shape than fiducial markers of the second plurality offiducial markers, wherein at least one fiducial marker of at least aportion of the first and second pluralities of fiducial markerscomprises a plurality of distinct regions, and wherein the plurality ofdistinct regions comprises at least two of: a first region having acrosshair that indicates a center of the fiducial marker, a secondregion having an angular barcode that uniquely identifies the fiducialmarker within a calibration image taken of the fiducial marker, a thirdregion that identifies a boundary of the fiducial marker, or a fourthregion having a human-readable label that uniquely identifies thefiducial marker within the calibration image taken of the fiducialmarker; determining locations and identifications of one or more of thefirst and second pluralities of fiducial markers in the calibrationimage; and based on the determined locations and identifications,calibrating the camera.
 8. The method of claim 7, further comprising:rotating the camera relative to the target and recording an additionalcalibration image; and determining locations and identifications of oneor more of the first and second pluralities of fiducial markers in theadditional calibration image.
 9. The method of claim 8, wherein rotatingthe camera relative to the target comprises rotating the camera about aroll axis of the camera.
 10. The method of claim 7, further comprising:moving the camera and recording an additional calibration image; anddetermining locations and identifications of one or more of the firstand second pluralities of fiducial markers in the additional calibrationimage.
 11. The method of claim 7, wherein the first pattern is a firstrectangular pattern, wherein the second pattern is a second rectangularpattern, and wherein the first rectangular pattern is scaled such thatthe first rectangular pattern has larger horizontal and verticaldimensions than the second rectangular pattern.
 12. The method of claim7, wherein the first region is circular in shape and approximatelycentered at a center of the fiducial marker, wherein the second regionis annular in shape and is positioned adjacent to a perimeter of thefirst region, and wherein the third region is annular in shape and ispositioned adjacent to a perimeter of the second region.
 13. A systemused for calibrating a camera, comprising: a target comprising a panelhaving a first plurality of fiducial markers arranged on the panel in afirst pattern and a second plurality of fiducial markers arranged on thepanel in a second pattern, wherein the first pattern is a scaled versionof the second pattern, such that a calibration image captured of thetarget simulates multiple images of a single pattern captured atmultiple calibration perspectives, and wherein fiducial markers of thefirst plurality of fiducial markers have a different shape than fiducialmarkers of the second plurality of fiducial markers, wherein at leastone fiducial marker of at least a portion of the first and secondpluralities of fiducial markers comprises a plurality of distinctregions, and wherein the plurality of distinct regions comprises atleast two of: a first region having a crosshair that indicates a centerof the fiducial marker, a second region having an angular barcode thatuniquely identifies the fiducial marker within a calibration image takenof the fiducial marker, a third region that identifies a boundary of thefiducial marker, or a fourth region having a human-readable label thatuniquely identifies the fiducial marker within the calibration imagetaken of the fiducial marker; and a stage configured to translate orrotate the camera with respect to the target.
 14. The system of claim13, wherein the first pattern is a first rectangular pattern, whereinthe second pattern is a second rectangular pattern, and wherein thefirst rectangular pattern is scaled such that the first rectangularpattern has larger horizontal and vertical dimensions than the secondrectangular pattern.
 15. The system of claim 13, wherein the firstregion is circular in shape and approximately centered at a center ofthe fiducial marker, wherein the second region is annular in shape andis positioned adjacent to a perimeter of the first region, and whereinthe third region is annular in shape and is positioned adjacent to aperimeter of the second region.
 16. The system of claim 13, wherein thepanel is the only panel the target comprises.
 17. The system of claim13, wherein the panel further comprises a panel-identification fiducialmarker that uniquely identifies the target.